Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] support more file/array schema in tools #292

Merged
merged 6 commits into from
Jan 16, 2024

Conversation

wj-Mcat
Copy link
Collaborator

@wj-Mcat wj-Mcat commented Jan 15, 2024

No description provided.

@wj-Mcat
Copy link
Collaborator Author

wj-Mcat commented Jan 15, 2024

在 APIHub 上面预支 tool 的执行结果:

============================= test session starts ==============================
platform darwin -- Python 3.10.13, pytest-7.4.2, pluggy-1.3.0 -- /Users/wujingjing05/miniconda3/envs/eb-agent/bin/python
cachedir: .pytest_cache
rootdir: /Users/wujingjing05/projects/yiyan/ERNIE-Bot-SDK-add-tool-unittest/erniebot-agent
plugins: cov-4.1.0, asyncio-0.23.2, anyio-4.2.0
asyncio: mode=strict
collecting ... collected 24 items

tests/integration_tests/apihub/test_car_cls.py::TestRemoteTool::test_car_classify PASSED [  4%]
tests/integration_tests/apihub/test_dish.py::TestRemoteTool::test_dish_classify PASSED [  8%]
tests/integration_tests/apihub/test_doc_rm_bnd.py::TestRemoteTool::test_doc_rm_bnd PASSED [ 12%]
tests/integration_tests/apihub/test_handwriting.py::TestRemoteTool::test_hand_text_rec PASSED [ 16%]
tests/integration_tests/apihub/test_high_ocr.py::TestRemoteTool::test_tool PASSED [ 20%]
tests/integration_tests/apihub/test_image_moderation.py::TestRemoteTool::test_tool PASSED [ 25%]
tests/integration_tests/apihub/test_img_enhance.py::TestRemoteTool::test_tool PASSED [ 29%]
tests/integration_tests/apihub/test_img_transform.py::TestRemoteTool::test_img_style_trans PASSED [ 33%]
tests/integration_tests/apihub/test_img_transform.py::TestRemoteTool::test_person_animation PASSED [ 37%]
tests/integration_tests/apihub/test_landmark.py::TestRemoteTool::test_dish_classify PASSED [ 41%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_formula FAILED [ 45%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_ocr_general PASSED [ 50%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_ocr_pp PASSED [ 54%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_pp_ocr_v4 PASSED [ 58%]
tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_shopping_receipt FAILED [ 62%]
tests/integration_tests/apihub/test_pic_translate.py::TestRemoteTool::test_tool FAILED [ 66%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_matting PASSED [ 70%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_ocr_v4 PASSED [ 75%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_structure PASSED [ 79%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_tinypose PASSED [ 83%]
tests/integration_tests/apihub/test_pp_models.py::TestPPRemoteTool::test_pp_vehicle PASSED [ 87%]
tests/integration_tests/apihub/test_text_moderation.py::TestRemoteTool::test_tool PASSED [ 91%]
tests/integration_tests/apihub/test_text_to_speech.py::TestRemoteTool::test_text_to_speech PASSED [ 95%]
tests/integration_tests/apihub/test_translation.py::TestRemoteTool::test_tool FAILED [100%]

=================================== FAILURES ===================================
_________________________ TestRemoteTool.test_formula __________________________

self = <tests.integration_tests.apihub.test_ocr.TestRemoteTool testMethod=test_formula>

    @pytest.mark.asyncio
    async def test_formula(self):
>       toolkit = RemoteToolkit.from_aistudio("formula", file_manager=self.file_manager)

tests/integration_tests/apihub/test_ocr.py:69: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
    return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
    toolkit = RemoteToolkit.from_openapi_dict(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'erniebot_agent.tools.remote_toolkit.RemoteToolkit'>
openapi_dict = {'errorCode': 500, 'errorMsg': '工具不存在', 'logId': '8b8c797e20df8dc01527aff20169f34c', 'servers': [{'url': 'https://tool-formula.aistudio-hub.baidu.com'}]}
access_token = '4ce50e3378f418d271c480c8ddfa818537071dbe'
file_manager = <erniebot_agent.file.file_manager.FileManager object at 0x106cc2470>

    @classmethod
    def from_openapi_dict(
        cls,
        openapi_dict: Dict[str, Any],
        access_token: Optional[str] = None,
        file_manager: Optional[FileManager] = None,
    ) -> RemoteToolkit:
>       info = EndpointInfo(**openapi_dict["info"])
E       KeyError: 'info'

src/erniebot_agent/tools/remote_toolkit.py:175: KeyError
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-71' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.467 seconds
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-71' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.228 seconds
_____________________ TestRemoteTool.test_shopping_receipt _____________________

self = <tests.integration_tests.apihub.test_ocr.TestRemoteTool testMethod=test_shopping_receipt>

    @pytest.mark.asyncio
    async def test_shopping_receipt(self):
>       toolkit = RemoteToolkit.from_aistudio("shopping-receipt", file_manager=self.file_manager)

tests/integration_tests/apihub/test_ocr.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
    return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
    toolkit = RemoteToolkit.from_openapi_dict(
src/erniebot_agent/tools/remote_toolkit.py:185: in from_openapi_dict
    parameter_view = ToolParameterView.from_openapi_dict(schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'erniebot_agent.tools.schema.ToolParameterView'>
schema = {'properties': {'log_id': {'type': 'integer'}, 'pdf_file_size': {'type': 'string'}, 'words_result': {'properties': {'c...d', 'pdf_file_size', 'words_result'], 'type': 'object', 'x-apifox-orders': ['log_id', 'pdf_file_size', 'words_result']}

    @classmethod
    def from_openapi_dict(cls, schema: dict) -> Type[ToolParameterView]:
        """parse openapi component schemas to ParameterView
        Args:
            response_or_returns (dict): the content of status code
    
        Returns:
            _type_: _description_
        """
    
        # TODO(wj-Mcat): to load Optional field
        fields = {}
        for field_name, field_dict in schema.get("properties", {}).items():
            # skip loading invalid field to improve compatibility
            if "type" not in field_dict:
                raise ToolError(f"`type` field not found in `{field_name}` property", stage="Loading")
    
            if "description" not in field_dict:
>               raise ToolError(f"`description` field not found in `{field_name}` property", stage="Loading")
E               erniebot_agent.utils.exceptions.ToolError: An error occured in stage <Loading>. The error message is `description` field not found in `log_id` property

src/erniebot_agent/tools/schema.py:250: ToolError
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-95' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 1.080 seconds
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-95' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.310 seconds
___________________________ TestRemoteTool.test_tool ___________________________

self = <tests.integration_tests.apihub.test_pic_translate.TestRemoteTool testMethod=test_tool>

    @pytest.mark.asyncio
    async def test_tool(self):
        toolkit = RemoteToolkit.from_aistudio("pic-translate", file_manager=self.file_manager)
    
        file = await self.file_manager.create_file_from_path(self.download_fixture_file("shouxiezi.png"))
        agent = self.get_agent(toolkit)
    
>       result = await agent.run("这张照片里面讲了啥?", files=[file])

tests/integration_tests/apihub/test_pic_translate.py:18: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/agents/agent.py:129: in run
    raise e
src/erniebot_agent/agents/agent.py:126: in run
    agent_resp = await self._run(prompt, files)
src/erniebot_agent/agents/function_agent.py:149: in _run
    curr_step, new_messages = await self._step(chat_history)
src/erniebot_agent/agents/function_agent.py:190: in _step
    tool_resp = await self.run_tool(tool_name=tool_name, tool_args=tool_args)
src/erniebot_agent/agents/agent.py:176: in run_tool
    raise e
src/erniebot_agent/agents/agent.py:173: in run_tool
    tool_resp = await self._run_tool(tool, tool_args)
src/erniebot_agent/agents/agent.py:251: in _run_tool
    tool_ret = await tool(**parsed_tool_args)
src/erniebot_agent/tools/remote_tool.py:139: in __call__
    tool_arguments = await self.__pre_process__(tool_arguments)
src/erniebot_agent/tools/remote_tool.py:117: in __pre_process__
    tool_arguments = self.tool_view.parameters(**tool_arguments).model_dump(mode="json")
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = OpenAPIParameterView(image=b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x03\x06\x00\x00\x03\xf8\x08\x02\x00\x00\x00\x...\xd8W\xe2\x8b\xe6k7c\xaeN+\x03.\xd7\xfe?eP\xf75\xb6\xa4\x95L\x00\x00\x00\x00IEND\xaeB`\x82', from='auto', to='zh', v=3)

    def model_dump(
        self,
        *,
        mode: Literal['json', 'python'] | str = 'python',
        include: IncEx = None,
        exclude: IncEx = None,
        by_alias: bool = False,
        exclude_unset: bool = False,
        exclude_defaults: bool = False,
        exclude_none: bool = False,
        round_trip: bool = False,
        warnings: bool = True,
    ) -> dict[str, Any]:
        """Usage docs: https://docs.pydantic.dev/2.5/concepts/serialization/#modelmodel_dump
    
        Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
    
        Args:
            mode: The mode in which `to_python` should run.
                If mode is 'json', the dictionary will only contain JSON serializable types.
                If mode is 'python', the dictionary may contain any Python objects.
            include: A list of fields to include in the output.
            exclude: A list of fields to exclude from the output.
            by_alias: Whether to use the field's alias in the dictionary key if defined.
            exclude_unset: Whether to exclude fields that have not been explicitly set.
            exclude_defaults: Whether to exclude fields that are set to their default value from the output.
            exclude_none: Whether to exclude fields that have a value of `None` from the output.
            round_trip: Whether to enable serialization and deserialization round-trip support.
            warnings: Whether to log warnings when invalid fields are encountered.
    
        Returns:
            A dictionary representation of the model.
        """
>       return self.__pydantic_serializer__.to_python(
            self,
            mode=mode,
            by_alias=by_alias,
            include=include,
            exclude=exclude,
            exclude_unset=exclude_unset,
            exclude_defaults=exclude_defaults,
            exclude_none=exclude_none,
            round_trip=round_trip,
            warnings=warnings,
        )
E       UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid utf-8

../../../../miniconda3/envs/eb-agent/lib/python3.10/site-packages/pydantic/main.py:308: UnicodeDecodeError
----------------------------- Captured stderr call -----------------------------
�[92mINFO - [Run][Start] FunctionAgent is about to start running with input:
�[94m这张照片里面讲了啥?�[92m�[0m
�[92mINFO - [LLM][Start] ERNIEBot is about to start running with input:
 role: �[94muser�[92m 
 content: �[94m这张照片里面讲了啥?
<file>file-local-da8582b4-b3a6-11ee-803b-eac7af941eda</file>�[92m �[0m
�[92mINFO - [LLM][End] ERNIEBot finished running with output:
 role: �[93massistant�[92m 
 function_call: �[93m
{
  "name": "pic-translate/v1.0/pic_translate",
  "thoughts": "用户想要知道图片中的文字内容,我需要调用OCR工具来识别图片中的文字,然后根据需要翻译成中文。任务拆解:[sub-task1: 使用OCR工具识别图片中的文字,sub-task2: 根据需要将文字翻译成中文]。接下来,我需要调用[pic-translate/v1.0/pic_translate]来翻译图片中的文字。",
  "arguments": "{\"image\":\"file-local-da8582b4-b3a6-11ee-803b-eac7af941eda\",\"to\":\"zh\"}"
}�[92m �[0m
�[92mINFO - [Tool][Start] �[95mRemoteTool�[92m is about to start running with input:
�[95m{
  "image": "file-local-da8582b4-b3a6-11ee-803b-eac7af941eda",
  "to": "zh"
}�[92m�[0m
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-98' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:101> wait_for=<Future pending cb=[shield.<locals>._outer_done_callback() at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/tasks.py:864, Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 1.090 seconds
___________________________ TestRemoteTool.test_tool ___________________________

self = <tests.integration_tests.apihub.test_translation.TestRemoteTool testMethod=test_tool>

    @pytest.mark.asyncio
    async def test_tool(self):
>       toolkit = RemoteToolkit.from_aistudio("translation", file_manager=self.file_manager)

tests/integration_tests/apihub/test_translation.py:13: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/erniebot_agent/tools/remote_toolkit.py:261: in from_aistudio
    return cls.from_url(tool_url, version=version, access_token=access_token, file_manager=file_manager)
src/erniebot_agent/tools/remote_toolkit.py:305: in from_url
    toolkit = RemoteToolkit.from_openapi_dict(
src/erniebot_agent/tools/remote_toolkit.py:185: in from_openapi_dict
    parameter_view = ToolParameterView.from_openapi_dict(schema)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'erniebot_agent.tools.schema.ToolParameterView'>
schema = {'properties': {'log_id': {'description': 'logId', 'type': 'integer'}, 'result': {'properties': {'from': {'description... '翻译结果数组列表', 'items': {'properties': {...}, 'type': 'object'}, 'type': 'array'}}, 'type': 'object'}}, 'type': 'object'}

    @classmethod
    def from_openapi_dict(cls, schema: dict) -> Type[ToolParameterView]:
        """parse openapi component schemas to ParameterView
        Args:
            response_or_returns (dict): the content of status code
    
        Returns:
            _type_: _description_
        """
    
        # TODO(wj-Mcat): to load Optional field
        fields = {}
        for field_name, field_dict in schema.get("properties", {}).items():
            # skip loading invalid field to improve compatibility
            if "type" not in field_dict:
                raise ToolError(f"`type` field not found in `{field_name}` property", stage="Loading")
    
            if "description" not in field_dict:
>               raise ToolError(f"`description` field not found in `{field_name}` property", stage="Loading")
E               erniebot_agent.utils.exceptions.ToolError: An error occured in stage <Loading>. The error message is `description` field not found in `result` property

src/erniebot_agent/tools/schema.py:250: ToolError
------------------------------ Captured log call -------------------------------
WARNING  asyncio:base_events.py:1904 Executing <Task pending name='Task-152' coro=<IsolatedAsyncioTestCase._asyncioLoopRunner() running at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:95> wait_for=<Future pending cb=[Task.task_wakeup()] created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/asyncio/base_events.py:429> created at /Users/wujingjing05/miniconda3/envs/eb-agent/lib/python3.10/unittest/async_case.py:117> took 0.271 seconds
=========================== short test summary info ============================
FAILED tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_formula
FAILED tests/integration_tests/apihub/test_ocr.py::TestRemoteTool::test_shopping_receipt
FAILED tests/integration_tests/apihub/test_pic_translate.py::TestRemoteTool::test_tool
FAILED tests/integration_tests/apihub/test_translation.py::TestRemoteTool::test_tool
=================== 4 failed, 20 passed in 237.27s (0:03:57) ===================

@wj-Mcat
Copy link
Collaborator Author

wj-Mcat commented Jan 15, 2024

以上报错的几个 tool 是由于 yaml 内容错误导致的。

也就是现在的 PR 能够兼容之前tool 的文件解析格式。

Copy link
Member

@Bobholamovic Bobholamovic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File部分LGTM

@wj-Mcat wj-Mcat merged commit 069d774 into PaddlePaddle:develop Jan 16, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants